Financial 

Management 

Implementation  of  the  Data  Quality 
Management  Control  Program  for 
the  Military  Health  System 

(D-2002-141) 


Department  of  Defense 
Office  of  the  Inspector  General 


Quality 


Integrity 


Accountability 


Report  Documentation  Page 


Report  Date  Report  Type  Dates  Covered  (from...  to) 

29  Aug  2002  N/A 

Title  and  Subtitle 

Financial  Management:  Implementation  of  the  Data 

Quality  Management  Control  Program  for  the  Military 
Health  System 

Contract  Number 

Grant  Number 

Program  Element  Number 

Author(s) 

Project  Number 

Task  Number 

Work  Unit  Number 

Performing  Organization  Name(s)  and  Address(es) 

OAIG-AUD  (ATTN:  AFTS  Audit  Suggestions)  Inspector 
General,  Department  of  Defense  400  Army  Navy  Drive 
(Room  801)  Arlington,  VA  22202-2884 

Performing  Organization  Report  Number 

D-2002-141 

Sponsoring/Monitoring  Agency  Name(s)  and 

Address(es) 

Sponsor/Monitor’s  Acronym(s) 

Sponsor/Monitor’s  Report  Number(s) 

Distribution/Availability  Statement 

Approved  for  public  release,  distribution  unlimited 

Supplementary  Notes 

Abstract 


Subject  Terms 


Report  Classification 

unclassified 

Classification  of  this  page 

unclassified 

Classification  of  Abstract 

unclassified 

Limitation  of  Abstract 

UU 

Number  of  Pages 

27 


Additional  Copies 

To  obtain  additional  copies  of  this  report,  visit  the  Web  site  of  the  Inspector 
General  of  the  Department  of  Defense  at  www.dodig.osd.mil/audit/reports  or 
contact  the  Secondary  Reports  Distribution  Unit  of  the  Audit  Followup  and 
Technical  Support  Directorate  at  (703)  604-8937  (DSN  664-8937)  or  fax  (703) 
604-8932. 

Suggestions  for  Future  Audits 

To  suggest  ideas  for  or  to  request  future  audits,  contact  the  Audit  Followup  and 
Technical  Support  Directorate  at  (703)  604-8940  (DSN  664-8940)  or  fax 
(703)  604-8932.  Ideas  and  requests  can  also  be  mailed  to: 

OAIG-AUD  (ATTN:  AFTS  Audit  Suggestions) 

Inspector  General  of  the  Department  of  Defense 
400  Army  Navy  Drive  (Room  801) 

Arlington,  VA  22202-4704 

Defense  Hotline 

To  report  fraud,  waste,  or  abuse,  contact  the  Defense  Hotline  by  calling 
(800)  424-9098;  by  sending  an  electronic  message  to  Hotline@dodig.osd.mil;  or  by 
writing  to  the  Defense  Hotline,  The  Pentagon,  Washington,  DC  20301-1900.  The 
identity  of  each  writer  and  caller  is  fully  protected. 


Acronyms 


ASD(HA) 

DQMCP 

MRHB 

MTF 

TMA 

TOPS 


Assistant  Secretary  of  Defense  (Health  Affairs) 
Data  Quality  Management  Control  Program 
Military  Retirement  Health  Benefits 
Military  Treatment  Facility 
Tricare  Management  Activity 
Tricare  Operational  Performance  Statements 


INSPECTOR  GENERAL 
DEPARTMENT  OF  DEFENSE 
400  ARMY  NAVY  DRIVE 
ARLINGTON,  VIRGINIA  22202-4704 


August  29, 2002 

MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  DEFENSE  (HEALTH 
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SUBJECT :  Report  on  the  hnplementation  of  the  Data  Quality  Management  Control 
Program  for  the  Military  Health  Systan  (Report  No.  D-2002-141) 


We  are  providing  this  report  for  your  information  and  use.  We  conducted  the 
audit  in  response  to  the  Chief  Financial  Officers  Act  of  1990  and  the  Federal  Financial 
Management  Act  of  1994.  We  considered  management  comments  on  a  draft  of  this 
report  when  preparing  the  final  report.  The  Assistant  Secretary  of  Defense  (Health 
Affairs)  comments  conformed  to  the  requiremoits  of  DoD  Directive  7650.3;  therefore, 
additional  comments  are  not  required. 
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(DSN  664-9109)  (dvincent@dodig.osd.mil)  or  Mr.  Walter  R.  Loder  at  (703)  604-9636 
(DSN  664-9636)  (wloder@dodi^.osd.mil).  See  Appendix  C  for  the  report  distribution. 
The  team  members  are  listed  inside  the  back  cover, 

David  K.  Steensma 
Deputy  Assistant  Inspector  General 
for  Auditing 
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(Project  No.  D2001FA-0097) 


Implementation  of  the  Data  Quality  Management  Control 
Program  for  the  Military  Health  System 


Executive  Summary 

Who  Should  Read  This  Report  and  Why?  DoD  health  care  managers  should  read  this 
report.  Military  health  care  data  is  used  for  a  variety  of  critical  purposes,  including 
managing  patient  care,  determining  the  optimal  health  care  system,  and  for  financial 
management.  These  managers  can  directly  impact  the  quality  of  the  data  collected. 

Background.  The  audit  was  performed  in  support  of  Public  Law  101-576,  the  “Chief 
Financial  Officers  Act  of  1990,”  and  Public  Law  103-356,  the  “Federal  Financial 
Management  Act  of  1994.”  This  audit  supports  our  annual  auditing  of  the  DoD  Military 
Retirement  Health  Benefits  Liability  and  DoD  Agency-Wide  Financial  Statements.  The 
DoD  Office  of  the  Actuary  calculated  the  military  retirement  health  benefits  liability  at 
$192.4  billion  on  the  FY  2000  DoD  Agency-Wide  Financial  Statements.  The  liability 
has  tripled  to  more  than  $580  billion  with  the  implementation  of  the  National  Defense 
Authorization  Act  for  FY  2001,  which  extends  medical  benefits  to  military  retirees  who 
are  medicare-eligible.  This  calculation  relied  significantly  on  health  care  inpatient  and 
outpatient  workload  data  compiled  from  the  military  treatment  facilities  and  purchased 
care  providers. 

Results.  The  military  treatment  facilities  visited  did  not  fully  implement  the 
November  29,  2000,  Assistant  Secretary  of  Defense  (Health  Affairs)  policy  guidance  for 
the  Data  Quality  Management  Control  Program.  The  facilities  did  not  adequately 
prepare  or  complete  the  Data  Quality  Management  Control  Review  List  and  the  Data 
Quality  Statement.  Although  the  Surgeons  General  were  briefed  on  the  results  of  the 
program,  they  had  not  emphasized  training  of  personnel  in  the  facilities  in  order  to  ensure 
data  accuracy  and  standardization.  Tricare  Management  Activity  needs  to  adequately 
implement  the  Data  Quality  Management  Control  Program  by  initiating  a  DoD 
Instruction  for  program  guidance.  Then,  if  the  Surgeons  General  monitor  the  program 
and  provide  training  to  the  program  participants,  the  program  should  provide  the  controls 
needed  to  ensure  that  medical  provider  data  are  timely  and  accurate.  See  the  Finding 
section  for  details  on  the  recommendations. 

Management  Comments.  The  Assistant  Secretary  of  Defense  (Health  Affairs) 
concurred  with  the  recommendations;  and  stated  that  guidance  was  issued  to  improve 
monitoring  and  compliance  with  the  goals  of  improved  data  quality.  Reviews  of  data 
quality  were  initiated,  a  Data  Quality  Working  Group  was  established,  the  class  length 
for  the  Data  Quality  Management  Control  Program  was  increased,  and  a  draft  DoD 
Instruction  on  the  Data  Quality  Management  Program  was  in  coordination.  Three  major 
changes  to  improve  compliance  with  the  program  were  revisions  to  the  review  list  and 
data  quality  commander’s  statement,  reporting  Military  Retirement  Trust  Fund  level  of 
performance,  and  inclusion  of  the  Data  Quality  Management  Control  Program  into 
Service  Medical  Inspector  General  and  Audit  Agency  Compliance  Program.  Additional 
comments  are  not  required.  See  the  finding  section  for  a  discussion  of  management 
comments  and  the  Management  comments  section  for  the  complete  text  of  the  comments. 
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Background 


Financial  Statement  Requirement.  Public  Law  101-576,  the  “Chief  Financial 
Officers  Act  of  1990,”  November  15,  1990,  as  amended  by  Public  Law  103-356, 
the  “Federal  Financial  Management  Act  of  1994,”  October  13,  1994,  requires 
DoD  to  prepare  annual  audited  financial  statements.  The  FY  2000  DoD  Agency- 
Wide  Financial  Statements  include  financial  statements  for  a  reporting  entity 
titled  “Other  Defense  Organizations-General  Funds.”  The  Office  of  Management 
and  Budget  Bulletin  No.  01-09,  “Form  and  Content  of  Agency  Financial 
Statements,”  September  25,  2001,  outlines  guidance  for  reporting  pensions,  other 
retirement  benefits,  and  other  post-employment  benefits. 

Military  Retirement  Health  Benefits  Liability.  One  of  the  major  components 
of  the  Other  Defense  Organizations  Nonfederal  Liabilities  not  covered  by 
budgetary  resources  is  the  Military  Retirement  Health  Benefits  (MRHB)  Liability. 
The  calculation  of  the  medical  treatment  facility  portion  of  the  MRHB  Liability 
requires  extraction  of  workload,  cost,  and  demographic  data  from  multiple 
databases  within  the  Military  Healthcare  System  and  DoD.  Data  were  also 
extracted  from  the  Composite  Health  Care  System,  Medical  Expense  Performance 
Reporting  System,  and  the  Defense  Enrollment  &  Eligibility  Reporting  System. 
Purchased  care  workload  is  measured  by  using  the  Government  share  of  the 
health  care  episode  or  prescription  cost.  The  MRHB  Liability  estimate  relies 
significantly  on  the  military  treatment  facility  (MTF)  expense,  reported  by  the 
Medical  Expense  Performance  Reporting  System.  The  Composite  Health  Care 
System  provides  outpatient  workload  data  to  the  MRHB  Liability.  The  military 
treatment  facility  workload  data  are  then  extracted  from  the  Composite  Health 
Care  System.  The  DoD  Office  of  the  Actuary  calculated  the  MRHB  Liability  as 
$192.4  billion  on  the  DoD  Agency-Wide  FY  2000  Financial  Statements.  It  is 
projected  that  the  Military  Retirement  Health  Benefit  Liability  has  more  than 
tripled  to  $580  billion  with  the  implementation  of  the  National  Defense 
Authorization  Act  for  FY  2001,  which  extends  medical  benefits  to  military 
retirees  who  are  medicare  eligible. 

Prior  Audit  Recommendation.  The  Inspector  General  of  the  Department  of 
Defense  Report  No.  99-127,  “Data  Supporting  the  FY  1998  DoD  Military 
Retirement  Health  Benefits  Liability  Estimate,”  April  7,  1999,  determined  that  the 
military  treatment  facility  workload  data  were  unreliable.  The  report  also  stated 
that  the  failure  to  verify  the  accuracy  of  the  workload  data  used  to  calculate  the 
MRHB  Liability  would  constitute  a  significant  impediment  in  achieving 
unqualified  opinions  on  both  DoD  and  Government- Wide  Financial  Statements. 
Additionally,  the  report  recommended  that  the  Assistant  Secretary  of  Defense 
(Health  Affairs)  (ASD[HA])  develop  and  implement  a  data  quality  assurance 
program.  In  response  to  the  audit  recommendation,  the  ASD(HA)  developed  the 
Data  Quality  Management  Control  Program  instituted  by  the  November  29,  2000, 
Policy  Memorandum. 
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Data  Quality.  An  integral  part  of  performance  measurement  for  the  military 
health  system  is  data  quality.  To  provide  the  right  information  to  the  right  people 
at  the  right  time  requires  accurate,  timely,  and  complete  data.  The  military  health 
system  cost  and  workload  reporting  systems  provide  data  that  are  used  for  the 
following  reasons: 

•  making  health  care  policy  decisions, 

•  preparing  cost  allocations, 

•  negotiating  with  managed  care  support  contractors, 

•  establishing  billing  rates, 

•  justifying  expenditures  and  budgets,  and 

•  supporting  third  party  collection  and  medicare  subvention  initiatives. 


Data  Quality  Management  Control  Program.  The  ASD(HA)  FY  2000  Annual 
Statement  of  Assurance  identified  military  health  system  data  quality 
management  controls  as  a  material  control  weakness.  The  ASD(HA)  Policy 
Memorandum,  “Data  Quality  Management  Control  Program,”  dated 
November  29,  2000,  establishes  the  Data  Quality  Management  Control  Program 
(DQMCP)  for  the  military  health  system.  The  DQMCP  was  developed  by  Tri- 
Service  working  groups  with  the  objective  of  improving  the  overall  quality  of 
MTF  financial  and  clinical  workload  data.  The  program  requires  command 
oversight  to  improve  the  submission  of  data  and  to  assure  the  uniformity  and 
standardization  of  the  information  across  the  Military  Health  System.  The  policy 
memorandum  recommends  the  following  reporting  organizational  structure. 

•  The  MTFs  are  to  complete  the  monthly  Data  Quality  Management  Control 
Review  List,  brief  the  MTF  Executive  Committee,  and  forward  the 
monthly  Commander’s  Statement  with  the  commander’s  signature. 

•  The  Service  data  quality  managers  are  to  identify  data  quality  deficiencies 
and  recommend  corrective  action,  brief  the  Military  Department  Surgeons 
General,  and  forward  data  quality  deficiencies  and  findings  with 
corrective  actions  proposed. 

•  The  Tricare  Management  Activity  should  subsequently  develop  metrics  to 
correct  and  improve  data  quality,  brief  Deputy  Surgeons  General,  and 
present  metrics  of  results  and  findings. 

The  ASD(HA)  provides  the  Annual  Statement  of  Assurance,  which  includes  the 
DQMCP,  to  the  Secretary  of  Defense.  The  program  was  designed  as  part  of  the 
overall  management  control  program  for  the  military  health  system.  The  DoD 
data  quality  characteristic  and  conformance  measures  are  accuracy,  completeness, 
consistency,  timeliness,  uniqueness,  and  validity.  See  Appendix  B  for  the 
DQMCP  flowchart. 
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Tricare  Management  Activity  (TMA)  Responsibility.  DoD  Directive  5136.12, 
“Tricare  Management  Activity,”  dated  May  31,  2001,  states  that  the  Director, 
TMA,  under  the  authority,  direction,  and  control  of  the  ASD(HA),  shall  manage 
the  execution  of  policy  in  the  administration  of  all  DoD  medical  and  dental 
programs.  TMA  issues  program  direction  for  the  execution  of  policy  within  the 
DoD  military  health  system  to  the  Surgeons  General  of  the  Army,  Navy,  and  Air 
Force.  For  the  DQMCP,  the  TMA  Resource  Management  Office  collects  the 
consolidated  data  from  the  Services,  disseminates  the  data,  and  produces  periodic 
reports.  TMA  Resource  Management  Office  is  also  responsible  for  compiling 
information  from  the  Commanders’  Statements  signed  by  the  MTF  Commanders 
and  corresponding  data  metrics  to  measure  program  performance.  The  TMA 
Resource  Management  Office  monitors  the  metrics  to  improve  data  quality  and  to 
implement  MTF-level  reporting  and  audit  oversight.  TMA  compiles  metrics 
submitted  by  the  three  Services,  then  submits  the  results  to  each  of  the  surgeon 
general  resource  managers  and  to  the  Deputy  Surgeons  General. 


Objectives 


The  objective  of  the  audit  was  to  determine  whether  the  MTFs,  the  Surgeons 
General,  and  the  TMA  had  implemented  the  Data  Quality  Management  Control 
Program  for  the  military  health  system.  Specifically,  we  determined  whether  the 
policies  and  procedures  described  in  the  November  29,  2000,  policy 
memorandum,  “Data  Quality  Management  Control  Program,”  issued  by  the 
ASD(HA)  were  implemented. 
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Data  Quality  Management  Control 
Program  for  the  Military  Health  System 

Military  treatment  facilities  did  not  fully  implement  the 
November  29,  2000,  ASD(HA)  policy  guidance  for  the  DQMCP. 
Specifically,  the  three  military  treatment  facilities  visited  did  not 
adequately  prepare  or  complete  the  Data  Quality  Management  Control 
Review  List  and  the  Data  Quality  Statement.  This  occurred  because  the 
ASD(HA)  did  not  provide  adequate  program  implementing  guidance  to 
initiate  the  program;  the  Surgeons  General  of  the  Military  Departments 
did  not  provide  specific  training  to  implement  the  DQMCP;  and  the  TMA 
did  not  adequately  develop  metrics  to  monitor  program  results.  As  a 
result,  there  are  still  concerns  about  the  data  used  to  support  the  Military 
Retirement  Health  Benefit  Liability  Estimate,  which  will  impact  DoD 
decision  making. 


Data  Quality  Program  Requirements 


The  ASD(HA)  policy  memorandum,  dated  November  29,  2000,  establishes  the 
elements  for  the  DQMCP  for  the  military  health  system.  The  elements  include 
the  following  10  steps. 

Military  Treatment  Facility  Responsibilities.  Steps  1  through  3  state  that  each 
MTF  will  designate  a  data  quality  manager  and  a  data  quality  assurance  team  who 
will  complete  the  monthly  Data  Quality  Management  Control  Review  List 
(Review  List)  and  Commander’s  monthly  Data  Quality  Statement.  The  Review 
List  is  a  multiple  item  questionnaire  used  as  an  internal  tool  to  assist  in 
identifying  and  correcting  financial  and  clinical  workload  data  problems.  The 
completed  Review  List  provides  information  for  the  completion  of  the 
Commander’s  monthly  Data  Quality  Statement  (Commander’s  Statement).  The 
Commander’s  Statement  acknowledges  responsibility  for  the  financial  and 
clinical  workload  data  reported  from  his  or  her  respective  MTF.  Once  the  Review 
List  is  completed,  the  MTF  data  quality  manager  briefs  the  results  to  each  MTF 
executive  committee.  Each  MTF  Commander  will  chair  the  executive  committee 
for  that  facility  and  will  review,  sign,  and  forward  the  Commander’s  Statement  to 
the  Service  data  quality  manager. 

Military  Surgeons  General  Responsibilities.  Steps  4  through  6  state  that  the 
Commanders’  Statements  are  analyzed  by  the  Service  data  quality  manager,  who 
will  then  identify  the  overall  data  quality  deficiencies  and  propose  corrective 
actions.  The  Service  data  quality  manager  briefs  the  Military  Department’s 
Surgeon  General  on  the  data  quality  deficiencies  and  proposed  corrective  actions. 
The  Service  data  quality  manager  then  forwards  data  quality  deficiencies  and 
proposed  corrective  actions  to  the  resource  management  steering  committee 
through  the  management  control  program  office  in  TMA  Resource  Management. 
The  resource  management  steering  committee  is  the  biweekly  meeting  of  Service 
budget  representatives  to  discuss  issues  and  concerns  on  the  Defense  Health 


4 


Program  Budget.  Data  quality  is  a  critical  part  of  this  process  due  to  the  fact  that 
inaccurate  data  results  in  inaccurate  reports  and  possible  loss  of  funds. 

Tricare  Management  Activity  Responsibilities.  Steps  7  through  10  establish 
the  TMA  Resource  Management  Office’s  responsibility  for  developing  metrics  to 
measure  performance  of  the  DQMCP.  Metrics  from  the  DQMCP  will  be  reported 
in  Tricare  Operational  Performance  Statements  (TOPS).  TOPS  is  both  a 
statement  and  an  evaluation  tool  providing  a  snapshot  of  the  performance  of  the 
military  health  system.  As  a  statement,  it  establishes  a  historical  record  of 
performance  that  can  be  used  to  identify  long-term  trends  in  performance.  As  an 
evaluation  tool,  it  is  a  compendium  of  routine  analyses  conducted  by  the  TMA 
Office  of  Health  Program  Analysis  and  Evaluation  in  its  routine  surveillance  of 
the  military  health  system.  Metrics  from  the  DQMCP  will  be  presented  in 
briefings  to  the  Deputy  Surgeons  General.  The  ASD(HA)  will  provide  an  Annual 
Statement  of  Assurance  to  the  Secretary  of  Defense  regarding  the  status  of  the 
DQMCP.  Appendix  B  is  a  DQMCP  flowchart. 


MTF  Program  Implementation 

Two  of  the  three  MTFs  visited  had  attempted  to  prepare  the  Review  List  and  the 
Commander’s  Statement,  however  inconsistent  interpretation  of  the  guidance 
caused  incorrect  responses.  The  following  table  summarizes  the  degree  to  which 
the  lists  were  inaccurate. 


MTF  Incorrect  Responses 

Military  Treatment 
Facility 

Commander’s  Statement 
(8  questions  total) 

Review  List 
(47  questions  total) 

Brooke  Army  Medical 
Center 

3  (37.5  percent) 

8  (17.0  percent) 

Naval  Medical  Center 
Portsmouth 

6  (75.0  percent) 

Not  prepared  by  MTF 

Wilford  Hall  Medical 
Center 

5  (62.5  percent) 

19  (40.4  percent) 

At  each  MTF  visited,  the  Commander’s  Statement  and  Review  List  had  numerous 
incorrect  responses.  Additional  incorrect  answers  may  still  be  unaccounted  for 
because  we  could  not  determine  the  accuracy  of  all  answers  due  to  insufficient 
supporting  documentation.  As  a  result,  TMA  and  the  Surgeons  General  will  not 
have  accurate  information  to  make  sound  business  decisions  for  improving  data 
quality.  The  findings  for  each  MTF  visited  follow. 

Brooke  Army  Medical  Center.  The  Data  Quality  Manager  and  the  Data  Quality 
Assurance  Team  for  Brooke  Army  Medical  Center  did  not  adequately  prepare  the 
Review  List  and  the  Commander’s  Statement.  Brooke  Army  Medical  Center  had 
complied  with  the  steps  prescribed  by  the  November  29,  2000,  policy 
memorandum,  however,  they  answered  8  of  47  Review  List  questions  incorrectly. 
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For  example,  question  number  1  on  the  Commander’s  Statement  and  the 
corresponding  question,  B.7,  on  the  Review  List  were  designed  to  track  adherence 
to  requirements  for  daily  end-of-day  processing  procedures.  Army  reworded  the 
question  on  the  Brooke  Medical  Center  Review  List  to  ask,  “Does  your  policy 
support  the  adherence  to  requirements  for  daily  end-of-day  processing  procedures 
by  all  clinics?”  The  interpretation  significantly  shifts  the  focus  from  procedural 
adherence  to  having  a  policy  supporting  adherence.  Further,  according  to  Brooke 
Army  Medical  Center  personnel,  no  formal  written  procedures  had  been 
prescribed  for  end-of-day  processing  procedures  at  Brooke  Army  Medical  Center. 
Therefore,  the  answer  to  the  question  as  reworded  should  have  been  “No”  but  it 
was  answered  “Yes.” 


Wilford  Hall  Medical  Center.  While  Wilford  Hall  Medical  Center  complied 
with  Steps  1  through  3  of  the  DQMCP  for  the  February  2001  reporting  period, 
they  did  not  adequately  prepare  the  Review  List  and  the  Commander’s  Statement 
(Program  Step  1).  The  Review  List  and  the  Commander’s  Statement  had 
numerous  incorrect  responses,  which  could  not  be  properly  validated  because  of 
the  lack  of  supporting  documentation.  For  example.  Commander’s  Statement 
question  number  6  and  Review  List  question  C.17,  which  call  for  completions  of 
monthly  financial  and  workload  reconciliation’s  of  Military  Expense  Performance 
Reporting  System,  were  answered  “Yes”  when  evidence  showed  that  the  answers 
should  have  been  “No.”  Additional  incorrect  answers  may  still  be  unaccounted 
for  because  we  could  not  determine  the  accuracy  of  all  answers  due  to  the  lack  of 
supporting  documentation.  When  supporting  documentation  was  requested,  the 
Data  Quality  Assurance  Team  stated  that  there  was  not  a  requirement  to  retain 
supporting  documentation. 

Naval  Medical  Center  Portsmouth.  Naval  Medical  Center  Portsmouth  did  not 
fully  implement  the  Data  Quality  Management  Control  Program  and  did  not 
complete  the  Data  Quality  Control  Review  List  as  required  by  the  ASD(HA) 
policy  memorandum  of  November  29,  2000.  Although  the  Data  Quality 
Assurance  Team  met  monthly,  they  struggled  with  interpretation  issues  and 
subsequently  submitted  information  that  was  incorrect.  As  previously  stated  the 
Review  List  was  not  completed  and  75  percent  of  the  responses  to  the 
Commander’s  Statement  were  incorrect.  The  estimated  date  to  begin  completion 
of  the  Review  List  was  July  2001. 


Surgeon  General  Program  Implementation 


Although  the  Surgeons  General  of  the  Military  Departments  and  TMA  complied 
with  the  applicable  steps  in  the  DQMCP,  neither  provided  specific  training  to 
implement  the  program  to  ensure  standardized  collection  of  data  and  accuracy  of 
the  data  submitted  by  MTF.  Training  for  each  MTF  staff  member  is  normally 
conducted  at  the  MTF  or  at  the  Service-level  and  is  funded  by  the  Military 
Departments.  The  policy  memorandum  did  not  include  a  training  requirement. 
TMA  does  include  the  DQMCP  in  a  three-day  training  course  that  is  conducted 
quarterly.  Initially,  however,  the  three-day  course  devoted  only  one  hour  to  the 
Review  List  and  Commander’s  Statement.  TMA  has  since  enhanced  the  three- 
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day  training  course  to  allot  up  to  three  hours  to  teaehing  the  Review  List  and  the 
Commander’s  Statement.  At  eaeh  MTF  visited,  the  data  quality  assurance  team 
struggled  with  interpretation  issues.  Training  would  eliminate  interpretation 
issues  thus  creating  more  standardized  results  for  the  Review  List  and 
Commander’s  Statement. 


TMA  Program  Implementation  and  Followup 


The  ASD(HA)  policy  memorandum  of  November  29,  2000,  did  not  provide 
adequate  implementing  guidanee  for  the  TMA,  the  Surgeons  General,  and  eaeh 
MTF  to  initiate  and  monitor  the  program.  Speeifically,  the  guidanee  did  not 
inelude  the  following: 

•  a  system  for  monitoring  the  implementation  of  the  program  and  the 
accuraey  of  answers  to  the  Review  List  and  Commander’s  Statement 
questions, 

•  standardized  procedures  for  eompleting  the  questions  on  the  Review 
List,  and  a  requirement  for  retaining  supporting  doeumentation,  and 

•  a  detailed  proeess  for  doeumenting  and  traeking  the  aecomplishment 
of  corrective  actions,  and  metrics  for  monitoring  the  results  of  the 
program. 

Program  Monitoring.  The  TMA  previously  did  not  have  an  adequate 
monitoring  system  in  plaee  to  ensure  that  the  DQMCP  was  fully  implemented. 
Specifieally,  TMA  did  not  monitor  all  Review  Lists  eompleted  by  each  MTF  and 
the  aecuracy  of  the  answers.  Without  aetively  monitoring  the  implementation  and 
complianee  of  the  data  quality  program  and  the  accuracy  of  the  data  in  the 
metrics,  TMA  will  receive  unreliable  data.  TMA  reeeives,  assimilates,  and 
reviews  data  from  the  Services  and  tracks  the  results  of  eomplianee  within  the 
data  metrics.  TMA  issued  a  poliey  memorandum  dated  October  17,  2001,  revising 
the  Review  List  and  the  Commander’s  Statement.  The  changes  request  that  two 
additional  Data  Quality  Management  Control  measures  be  adopted  by  the 
Serviees  to  allow  TMA  better  oversight  of  data  quality  reporting  and  monitoring. 
However,  sinee  December  2001,  each  Service  Inspector  General  has  been 
required  to  eonduct  data  quality  audits  for  eomplianee  with  TMA  data  quality 
program  guidance.  Sinee  December  2001,  the  Service  data  quality  managers  in 
each  MTF  have  submitted  performance  reports  direetly  to  TMA  in  order  to  allow 
TMA  to  better  monitor  the  DQMCP.  Without  monitoring  the  implementation  of 
the  program  and  the  accuracy  of  the  answers,  TMA  could  receive  unreliable  data. 
Also,  military  health  system  data  quality  management  eontrols  will  continue  to  be 
reported  in  the  DoD  Annual  Statement  of  Assuranee  as  a  material  weakness. 

Standardized  Procedures.  At  eaeh  MTF  visited,  the  data  quality  assurance  team 
used  different  proeedures  to  answer  the  same  questions.  For  example,  to  answer 
question  C.12.A,  whieh  required  a  random  review  of  outpatient  appointments,  one 
MTF  used  an  existing  report  and  did  not  sample  any  reeords.  This  MTF  reported 
that  24  percent  of  the  outpatient  medical  records  could  not  be  located.  A  seeond 
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MTF  used  a  sample  of  32  records  selected  from  only  one  of  more  than  70  clinics. 
This  second  MTF  reported  that  78  percent  of  the  records  could  not  be  located. 
Available  guidance  did  not  specify  how  a  random  sample  should  be  conducted. 
This  use  of  inconsistent  sampling  methods  resulted  in  data  that  cannot  be  used  for 
comparison.  Support  documentation  was  not  available  to  validate  most  of  the 
responses  in  the  Review  List  or  Commander’s  Statement  at  any  of  the  three  MTFs 
visited.  When  supporting  documentation  was  requested,  data  quality  assurance 
teams  stated  that  there  was  not  a  requirement  to  retain  supporting  documentation. 
Specific  answers  to  questions  on  the  Review  List  and  Commander’s  Statements 
were  not  supported  at  two  of  the  three  MTFs  visited.  The  Review  List  was  not 
completed  at  the  third  MTF.  An  audit  trail  would  assist  all  levels  of  management 
in  ensuring  that  questions  were  answered  accurately  and  consistently.  In  response 
to  audit  efforts,  TMA  issued  a  policy  memorandum  on  October  17,  2001.  The 
memorandum  revises  the  Review  List  and  the  Commander’s  Statement,  and 
initiates  military  treatment  facility-level  reporting  to  TMA  beginning  December 
2001.  The  new  guidance  states  that,  for  tracking  purposes,  the  completed  forms 
and  accompanying  working  papers  must  be  kept  on  file  for  five  years.  The  policy 
memorandum  of  October  17,  2001,  also  includes  the  Data  Quality  Management 
Control  Program  in  each  Service’s  Medical  Inspector  General  compliance 
program. 

Metric  Development.  TMA  previously  did  not  establish  metrics  for  measuring 
program  effectiveness.  TMA  was  responsible  for  developing  metrics  to  measure 
the  results  of  the  program  and  for  posting  the  metrics  on  the  TMA  data  quality 
web  site.  At  the  time  of  our  review,  TMA  had  not  determined  the  specific  metrics 
for  the  data  quality  program.  Management  stated  they  were  waiting  for  initial 
results  before  deciding  on  appropriate  metrics.  We  believe  performance  measures 
should  be  determined  in  advance  of  program  implementation  in  order  to  establish 
management  expectations  and  goals.  TMA  was  unable  to  provide  the  Surgeons 
General  and  each  MTF  with  adequate  feedback  on  the  condition  of  data  quality 
without  the  appropriate  metrics.  As  of  October  16,  2001,  TMA  had  developed  a 
series  of  metrics  to  measure  the  effectiveness  of  the  Services’  documentation  of 
data  quality.  In  December  2001,  the  TMA  began  receiving  monthly  updated 
metrics  performance  data  from  each  individual  MTF.  The  TMA  plans  to  discuss 
the  results  with  the  Data  Quality  Management  Control  Workgroup,  which 
includes  representatives  from  TMA  and  the  services.  However,  the  metrics  are 
not  currently  posted  in  the  TOPS  on  the  TMA  Website. 

Tracking  Corrective  Actions.  Each  MTF  is  responsible  for  identifying  data 
quality  deficiencies  and  proposing  corrective  actions.  However,  the  policy 
memorandum  of  November  29,  2000,  does  not  establish  a  systematic  process  to 
document  that  corrective  actions  have  been  taken.  Without  tracking  the 
accomplishment  of  corrective  actions,  management  will  not  be  aware  of  any 
resolution  to  the  deficiencies  in  data  quality.  The  policy  memorandum  of 
October  17,  2001,  initiated  reporting  MTF-level  performance  to  TMA  by  the 
Service  data  quality  managers  beginning  with  the  December  2001  reporting 
period.  This  reporting  will  allow  TMA  better  oversight  of  DQMCP  issues  and 
improve  the  reporting  and  monitoring  of  the  DQMCP  overall. 


8 


DQMCP  Guidance 


As  previously  stated,  the  November  29,  2000,  ASD(HA)  policy  memorandum, 
“Data  Quality  Management  Control  Program,”  was  issued  for  the  purpose  of 
implementing  the  DQMCP.  The  original  policy  memorandum  fits  the  criteria  for 
a  directive-type  memorandum  as  explained  by  DoD  Directive  5025.1,  “DoD 
Directives  System,”  dated  July  27,  2000.  DoD  Directive  5025.1  states  that  a 
directive-type  memorandum  issued  by  the  Secretary  or  Deputy  Secretary  of 
Defense,  which  because  of  time  constraints  cannot  be  published  in  the  DoD 
Directives  System  at  the  time  of  signature,  must  be  converted  into  a  DoD 
Instruction  within  180  days  of  signature.  TMA  and  ASD(HA)  did  not  comply 
with  this  Directive.  TMA  has  issued  a  second  policy  memorandum  dated 
October  17,  2001,  revising  the  Review  List  and  the  Commander’s  Statement.  The 
second  policy  memorandum  revises  the  Review  Lists  and  provides  additional 
DQMCP  measures  to  be  adopted.  However,  the  second  policy  does  not  supersede 
the  first,  and  both  were  issued  after  July  27,  2000.  Therefore,  in  accordance  with 
DoD  Directive  5025.1,  the  policy  should  have  been  converted  into  a  DoD 
Instruction  by  January  28,  2001.  A  DoD  Instruction  is  a  policy  document 
containing  what  is  required  by  legislation  or  the  Secretary  of  Defense  to  initiate, 
govern,  or  regulate  actions  by  DoD  Components  within  their  specific  areas  of 
responsibility.  A  DoD  Instruction  will  regulate  and  ensure  the  implementation  of 
the  DQMCP. 


Effects  on  Data  Quality  Management  Control  Program 


The  Review  Lists  and  Commander’s  Statements  at  each  MTF  visited  were 
inaccurate  and  without  support  documentation  to  validate  responses.  As  a  result, 
TMA  was  provided  with  incorrect  and  incomplete  data  for  decision  making  on 
improving  data  quality.  If  corrective  actions  are  not  taken,  the  quality  of  the  data 
will  not  improve.  Specifically,  financial  and  workload  data  used  for  important 
management  decisions,  such  as  cost  allocation  and  cost  finding,  will  remain 
inaccurate.  As  a  result,  DoD  will  not  be  able  to  make  sound  management 
decisions,  the  MRHB  Liability  estimate  will  continue  to  be  unreliable  and 
military  health  system  data  quality  management  controls  will  continue  to  be 
reported  as  a  material  weakness  in  the  DoD  Annual  Statement  of  Assurance. 
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Recommendations,  Management  Comments,  and  Audit 
Response 


1.  We  recommend  that  the  TRICARE  Management  Activity  provide 
implementing  guidance  to  initiate  and  monitor  the  Data  Quality 
Management  Control  Program.  The  guidance  should  include: 

a.  a  system  for  monitoring  the  Implementation  of  the  program  and 
the  accuracy  of  answers  to  the  Review  List  and  Commander’s  Statement, 

b.  standardized  procedures  for  completing  the  questions  on  the 
Review  List, 

c.  a  requirement  for  retaining  supporting  documentation, 

d.  a  Service-Wide  detailed  process  for  documenting  and  tracking  the 
accomplishment  of  corrective  actions,  and 

e.  metrics  for  measuring  program  performance. 

Management  Comments.  The  TRICARE  Management  Activity  concurred  with 
the  recommendation.  The  Office  of  the  Assistant  Secretary  of  Defense  (Health 
Affairs)  requested  that  the  Services  add  the  Data  Quality  Management  Control 
Program  to  their  Service’s  Medical  IG  and/or  Audit  Agency  compliance  program 
and  is  requiring  that  Military  Treatment  Facilities’  level  of  performance  be 
reported  to  TRICARE  Management  Activity.  The  Assistant  Secretary  also 
requested  that  each  Service  develop  a  uniform  “Desk  Guide”  that  supports  and 
augments  uniformity  in  use  of  the  Review  List  and  Commander’s  Statement. 
Documentation  supporting  the  Review  List  and  Commander’s  Statement  will  be 
kept  on  file  for  5  years.  In  addition,  systemic  issues  and  Service-wide  facility- 
level  issues  identified  in  the  monthly  Review  List  and  Commander’s  Statement 
submissions  will  be  discussed,  documented,  and  tracked  at  periodic  TRICARE 
Management  Activity  Data  Quality  meetings.  Metrics  will  be  incorporated  and 
used  by  TRICARE  Management  Activity  to  measure  program  performance. 

2.  We  recommend  that  the  TRICARE  Management  Activity  comply  with 
DoD  Directive  5025.1,  “DoD  Directive  System,”  and  convert  the  policy 
memorandum  of  November  29,  2000,  and  all  subsequent  policy 
memorandums  on  the  implementation  of  the  Data  Quality  Management 
Control  Program  into  a  DoD  Directive. 

Management  Comments.  TRICARE  Management  Activity  concurred  with  the 
recommendation.  TRICARE  Management  Activity  submitted  a  Department  of 
Defense  Instruction  for  signature  by  the  Assistant  Secretary  of  Defense  (Health 
Affairs).  Future  memos  concerning  the  Data  Quality  Management  Control 
Program  will  be  incorporated  into  the  existing  Department  of  Defense  Instruction 
on  a  periodic  basis. 
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3.  We  recommend  that  the  Surgeons  General  of  the  Military  Departments: 

a.  monitor  the  compliance  with  and  execution  of  the  policies  and 
procedures  described  in  the  November  29,  2000,  policy  memorandum,  "Data 
Quality  Management  Control  Program,"  issued  by  the  Assistant  Secretary  of 
Defense  (Health  Affairs);  and 

b.  provide  specific  training  to  program  participants  that  will  secure 
compliance. 

Management  Comments.  TRICARE  Management  Aetivity  eoncurred  with  the 
reeommendation.  The  Offiee  of  the  Assistant  Seeretary  of  Defense,  Health 
Affairs  outlined  major  ehanges  in  the  Data  Quality  Management  Control  Program 
to  improve  military  treatment  faeilities’  eomplianee  with  the  program.  These 
include: 

•  changes  in  the  Review  List  and  Data  Quality  Commander’s  Statement, 

•  reporting  Military  Treatment  Facility-level  performance  to  TRICARE 
Management  Activity,  and 

•  inclusion  of  the  Data  Quality  Management  Control  Program  into  the 
Services’  Medical  IG  and/or  Audit  Agency  compliance  programs. 

In  addition,  the  portion  of  the  quarterly  TRICARE  Management  Activity  Data 
Quality  Training  course  that  is  specifically  devoted  to  Data  Quality  Management 
Control  Program  instruction  for  data  quality  personnel  has  been  increased  from  45 
minutes  to  3  hours. 

4.  We  recommend  that  the  military  treatment  facilities  comply  with  the 
policy  memorandum  guidance  pertaining  to  the  Data  Quality  Management 
Control  Program  by  accurately  completing  the  Data  Quality  Management 
Control  Review  List  and  the  Data  Quality  Statement. 

Management  Comments.  TRICARE  Management  Activity  concurred  with  the 
recommendation.  The  Office  of  the  Assistant  Secretary  of  Defense  (Health 
Affairs)  outlined  major  changes  in  the  Data  Quality  Management  Control 
Program,  which  should  improve  military  treatment  facilities’  compliance  with  the 
program. 
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Appendix  A.  Scope  and  Methodology 

Scope 


Work  Performed.  This  audit  focused  on  implementation  of  DQMCP  for  the 
military  health  system.  Specifically,  we  reviewed  MTF  implementation  of  the 
program  to  determine  whether  the  policies  and  procedures  described  in  the 
November  29,  2000,  policy  memorandum,  “Data  Quality  Management  Control 
Program,”  issued  by  the  ASD(HA)  were  implemented.  We  visited  TMA  and 
Office  of  the  Surgeon  General  for  each  Service  to  review  their  roles  in 
implementing  the  program. 

We  conducted  our  audit  at  the  following  DoD  MTF  sites: 

•  Brooke  Army  Medical  Center,  San  Antonio,  Texas; 

•  Wilford  Hall  Medical  Center,  San  Antonio,  Texas;  and 

•  Portsmouth  Naval  Medical  Center,  Portsmouth,  Virginia. 

Of  the  143  DoD  MTFs,  we  judgmentally  selected  those  three  for  review  from 
May  2001  to  January  2002  to  provide  audit  coverage  for  each  of  the  Military 
Departments.  The  three  MTFs  were  also  judgmentally  selected  because  of  the 
material  volume  of  outpatient  visits  at  those  facilities.  According  to  the 
ASD(HA),  the  three  MTFs  we  visited  reported  a  total  of  3.2  million  outpatient 
visits  (10  percent)  out  of  the  32.4  million  outpatient  visits  reported  for  143  MTFs 
worldwide  in  FY  2000. 

At  each  MTF  visited,  we  reviewed  procedures  for  implementing  the  DQMCP. 
Specifically,  we  reviewed  procedures  for  completing  the  Data  Quality 
Management  Control  Review  List  and  the  Commander’s  Statement. 

General  Accounting  Office  High-Risk  Area.  The  General  Accounting  Office 
has  identified  several  high-risk  areas  in  the  DoD.  This  report  provides  coverage 
of  the  Defense  Financial  Management  high-risk  area. 


Methodology 


Use  of  Computer-Processed  Data.  We  relied  on  computer-processed  data  to 
determine  whether  the  DQMCP  for  the  military  health  system  was  implemented. 
Specifically,  we  relied  on  workload  data  outputs  that  we  could  find  to  support 
documentation  for  the  answers  given  to  questions  on  the  Data  Quality 
Management  Control  Review  List  and  the  Data  Quality  Statement  at  each  MTF. 
We  did  not  validate  the  reliability  of  the  computer-processed  data  because  we 
limited  our  use  of  the  data  to  specific  questions  so  that  we  could  determine  the 
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adequacy  of  the  implementation  of  the  DQMCP  at  each  MTF  visited.  However, 
not  validating  the  reliability  of  MTF  computer-processed  data  did  not  materially 
affect  the  results  of  the  audit. 

Audit  Dates  and  Standards.  We  performed  this  audit  from  April  2001  to 
March  2002  in  accordance  with  generally  accepted  government  auditing 
standards. 

Contacts  During  the  Audit.  We  visited  or  contacted  organizations  within  DoD. 
Further  details  are  available  on  request. 


Prior  Coverage 


The  General  Accounting  Office  and  the  Inspector  General  of  the  Department  of 
Defense  (IG  DoD)  have  conducted  multiple  reviews  related  to  financial  statement 
issues.  General  Accounting  Office  reports  can  be  accessed  on  the  Internet  at 
http://www.gao.gov.  IG  DoD  reports  can  be  accessed  on  the  Internet  at 
http://www.dodig.osd.mil/  audit/reports.  The  IG  DoD  has  issued  four  reports  on 
the  military  retirement  health  benefits  liability  estimate  and  a  report  on  the 
Defense  Enrollment  Eligibility  Reporting  System  database. 

Inspector  General  of  the  Department  of  Defense 

IG  DoD  Report  No.  D-2001-154,  “Beneficiary  Data  Supporting  the  DoD 
Military  Retirement  Health  Benefits  Liability  Estimate,”  July  5,  2001 

IG  DoD  Report  No.  D-2000-194,  “Demographic  Data  Supporting  the  DoD 
Military  Retirement  Health  Benefits  Liability  Estimate,”  September  29, 
2000 

IG  DoD  Report  No.  D-2000-090,  “Inpatient  Data  Supporting  the  DoD 
Military  Retirement  Health  Benefits  Liability  Estimate,”  March  1,  2000 

IG  DoD  Report  No.  CIP02000S001,  “Evaluation  of  the  Criminal 
Investigative  Environment  in  which  the  Defense  Enrollment  Eligibility 
Reporting  System  Operates,”  January  7,  2000 

IG  DoD  Report  No.  99-127,  “Data  Supporting  the  FY  1998  DoD  Military 
Retirement  Health  Benefits  Liability  Estimate,”  April  7,  1999 
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The  Service  DQ  Manager  is  responsH:)le  for  consulting  with  the  MTF  DQ  Manager  to  implement  and  monitor  the  DQMC  Program 
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Appendix  C.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  (Comptroller)/Chief  Financial  Officer 
Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Under  Secretary  of  Defense  for  Personnel  and  Readiness 
Assistant  Secretary  of  Defense  (Health  Affairs) 


Department  of  the  Army 

Assistant  Secretary  of  the  Army  (Financial  Management  and  Comptroller) 
Surgeon  General,  Department  of  the  Army 
Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Naval  Inspector  General 

Surgeon  General,  Department  of  the  Navy 

Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Surgeon  General,  Department  of  the  Air  Force 
Auditor  General,  Department  of  the  Air  Force 


Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 


Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 
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Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member  (cont’d) 

House  Subcommittee  on  Government  Efficiency,  Financial  Management,  and 
Intergovernmental  Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  National  Security,  Veterans  Affairs,  and  International  Relations, 
Committee  on  Government  Reform 

House  Subcommittee  on  Technology  and  Procurement  Policy,  Committee  on 
Government  Reform 
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Assistant  Secretary  of  Defense  (Health  Affairs) 
Comments 


THE  ASSISTANT  SECRETARY  OF  DEFENSE 
WASHINGTON,  D.  C.  2030M200 


MAY  3  1  2002 


MEMORANDUM  FOR  DIRECTOR,  FINANCE  AND  ACCOUNTING  DIRECTORATE 
DEPARTMENT  OF  DEFENSE  INSPECTOR  GENERAL 

SUBJECT:  Draft  Report  on  Audit  “Implementation  of  the  Data  Quality  Management  Control 
Program  for  the  Military  Health  System”  (Project  No.  D2001 FA-0097) 


Thank  you  for  the  opportunity  to  review  and  provide  comments  on  the  draft  report 
“Implementation  of  die  Data  Quality  Management  Control  Program  for  the  Military  Health 
System.** 

Overall,  we  agree  with  the  findings  and  recommendations  contained  in  the  draft  report  to 
help  strengthen  the  Military  Health  System’s  Data  Quality  Management  Conrtol  Program, 
Attached  are  our  comments  on  the  discussion  draft  and  responses  to  the  recommendations. 

Please  feel  free  to  direct  any  questions  to  my  project  officers  on  this  effort,  Mr.  Terry 
Byce  (Functional)  at  (703)  681-6918  and  Mr.  Gunther  J.  Zimmerman  (GAO/IG  Liaison)  at  (703) 
681-7889, 

William  Winkenwerder,  Jr.,  MD 


Attachment: 
As  stated 
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DoD  IG  DRAFT  REPORT  -  DATED  APRIL  1, 2002 
(D2001FA-0097) 

Agency  Comments  on  Draft  Report,  ^Implementation  of  the  Data  Quality  Management 
Control  Program  for  the  Military  Health  System” 

DEPARTMENT  OF  DEFENSE  COMMENTS 


RECOMMENDATION  1.;  We  recommend  that  the  TRICARE  Management  Activity  provide 
implementing  guidance  to  initiate  and  monitor  the  Data  Quality  Management  Control  ^ogram. 
The  guidance  should  include: 

a.  A  system  for  monitoring  the  implementation  of  the  program  and  accuracy  of  answers 
to  the  Review  List  and  Commander’s  Statement, 

b.  Standardized  procedures  for  completing  the  questions  on  the  Review  List, 

c.  A  requirement  for  retaining  supporting  documentation, 

d.  A  Service-wide  detailed  process  for  documenting  and  tracking  the  accomplishment  of 
corrective  actions,  and 

e.  Metrics  for  measuring  program  performance. 

DoD  RESPONSE;  The  following  is  the  TRICARE  response  to  paragraphs  a-e  in 
Recommendation  1: 

a.  The  TRICARE  Management  Activity  (TMA)  concurs  with  this  reconmiendation.  As 
stated  above  in  the  subject  report,  the  October  17, 2001  Data  Quality  Memorandum 
signed  by  Mr.  Garrato  requests  that  the  Services  add  the  DQMC  Program  to  their 
Service’s  Medical  IG  and/or  Audit  Agency  compliance  program  along  with  requiring 
the  reporting  of  MTF  level  of  performance  to  TMA.  This  will  greatly  improve  the 
reporting  and  monitoring  of  this  important  program.  The  results  of  these  Service 
initiated  audits/reviews  will  be  discussed  at  the  monthly  Data  Management  Control 
Work  Group  meetings, 

b.  The  TRICARE  Management  Activity  (TMA)  concurs  with  this  recommendation.  As 
stated  above  in  the  subject  report,  the  October  17, 2001  Data  (Juality  Memorandum 
signed  by  Mr.  Carrato  outlined  the  changes  in  the  Review  List  and  the  Data  Quality 
Commander’s  Statement  which  affect  this  line  item.  The  Review  List  was  reduced  in 
size  from  47  questions  to  29  questions.  The  Data  Quality  Commander’s  Statement 
was  reduced  from  8  questions  to  7.  Each  Service  also  has  developed  a  “Desk  Guide” 
that  sui^orts  and  augments  the  use  of  the  Review  List  and  Commander’s  Statement. 
This  guide  is  shared  with  the  other  Services  and  their  Data  Quality  POCs  to  insure 
uniform  guidance  and  standards.  Another  aspect  of  our  response  to  this  question  is 
addressed  in  the  response  to  Recommendation  3b  concerning  training.  Our  goal  in 
DQMC  training  is  to  issue  a  coordinated,  integrated,  multi-Service  response  to  Data 
Quality  issues  concerning  the  Review  List  and  Commander’s  Statement  not  only  in 
the  training  provided  but  also  in  the  responses  provided  to  questions  from  the 
attendees. 
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c.  The  TRICARE  Management  Activity  (TMA)  concurs  with  this  recommendation.  As 
stated  above  in  the  subject  report,  the  October  17, 2001  Data  Quality  Memorandum 
signed  by  Mr.  Garrato  outlined  the  changes  in  the  Review  List  and  the  Data  Quality 
Commander’s  Statement  which  affect  this  line  item.  The  completed  forms  and  work 
papers  for  this  program  must  be  kept  on  file  for  5  years. 

d.  The  TRICARE  Management  Activity  (TMA)  concurs  with  this  recommendation. 
Currently,  TMA  Resource  Management  Directorate  tracks  TMA-wide  systemic 
issues  and  the  Services  are  required  to  track  their  respective  MTF  level  issues  and 
discrepancies  that  surface  from  the  monthly  Review  List  and  Data  (Juality 
Commander’s  Statement  submissions.  These  issues  are  discussed  and  documented  at 
the  Monthly  TMA  Data  Quality  meetings  and  TMA-wide  issues  are  briefed  to  the 
Resource  Management  Steering  Committee  (monthly)  and  Deputy  Surgeons  General 
(quarterly). 

e.  The  TRICARE  Management  Activity  (TMA)  concurs  with  this  recommendation. 
Current  metrics  being  utilized  by  TMA  to  measure  compliance  are  based  on  the  Data 
Quality  Commander’s  Statement  with  95-100  (GREEN),  80-94  (YELLOW),  and  less 
than  80  (RED),  These  metrics  have  been  utilized  in  briefings  to  the  RMSC,  DSGs, 
and  TMA  Executive  Director  and  have  been  the  nucleus  for  change  in  the  areas  of 
MEPRS  Reporting/Reconciliation  and  inpatient/outpatient  coding  initiatives.  These 
metrics  are  now  posted  (at  the  MTF  level  of  detail)  monthly  on  the  Data  Quality  Web 
Page,  TMA-RM  Web  Page,  and  quarterly  in  TRICARE  Operational  Performance 
Statement  (TOPS).  The  Data  Quality  Work  Group  is  also  looking  at  other  validations 
of  these  metrics,  audit/review  work  done  by  external  organizations,  and  control  tools 
used  by  other  functional  work  groups,  for  example,  the  MEPRS  (Medical  Expense 
Performance  Reporting  System)  Early  Warning  and  Control  System  (MEWACS/Web 
Based). 


RECOMMENDATION  2.;  We  recommend  that  the  TRICARE  Management  Activity  comply 
with  DoD  Directive  5025.1,  “DoD  Directive  System”  and  convert  the  policy  memorandum  of 
November  29, 2(X)0,  and  all  subsequent  policy  memorandum  on  the  implementing  of  the  data 
Quality  Management  Control  Program  into  a  DoD  Instruction, 

DoD  RESPONSE;  The  TRICARE  Management  Activity  (TMA)  concurs  with  this 
recommendation.  TMA  submitted  a  DoDI  for  signature  by  Dr,  William  Winkenwerder,  ASD 
(HA).  This  DoDI  is  currently  undergoing  coordination  and  includes  all  policy  memos  since 
inception  of  the  Data  Quality  Management  Control  Program.  Future  memos  concerning  the 
program  will  be  incorporated  into  the  DoDI  through  a  change  to  the  existing  DoDI  on  a  periodic 
basis. 


RECOMMENDATION  3.:  We  recommend  that  the  Surgeons  General  of  the  Military 
Departments: 

a.  Monitor  the  compliance  with,  and  execution  of,  the  policies  and  procedures  described 
in  the  November  29, 2(X)0,  policy  memorandum,  “Data  (Juality  Management  Control 
Program,”  issued  by  the  Assistant  Secretary  of  Defense  (Health  Affairs);  and 
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b.  Provide  specific  training  to  program  participants  that  ensure  accountability. 

DoD  RESPONSE;  The  following  is  the  TRICARE  response  to  paragraphs  a-b  in 
Recommendations: 

a.  The  TRICARE  Management  Activity  (TMA)  concurs  with  this  recommendation.  As 
stated  above  in  the  subject  report,  the  October  17, 2001  Data  Qudity  Memorandum 
signed  by  Mr.  Carrato  outlined  the  changes  in  the  Review  List,  Data  Quality 
Commander’s  Statement,  and  program  compliance  which  affect  this  line  item.  This 
memo  was  the  result  of  interim  input  from  the  DoDIG  from  their  visits  to  major 
medical  centers  in  the  three  Services.  Mr.  Carrato  outlined  three  major  changes  in  the 
program  to  improve  MTF  compliance  with  the  program.  These  were; 

1.  Changes  in  the  Review  Ust  and  Data  Quality  Commander’s  Statement, 

2.  Reporting  MTF  level  performance  to  TMA, 

3.  And  inclusion  of  DQMC  Program  into  Service  Medical  IG  and  /or  Audit 
Agency  compliance  program. 

These  three  items  apply  to  both  the  Service  and  the  MTF  as  you  can  see  below  in  the 
response  to  Recommendation  #4. 

b.  The  TRICARE  Management  Activity  (TMA)  concurs  with  this  recommendation. 

The  Services  are  currently  sending  personnel  that  are  in  “Data  Quality”  positions  to 
the  quarterly  TMA  Data  Quality  Class  held  in  Arlington,  Virginia.  The  section  of  this 
class  devoted  to  the  Data  Quality  Management  Control  Program  has  been  increased 
from  45  minutes  to  3  hours  based  on  interim  guidance  from  the  DoDIG  who  was 
invited  to  attend  an  earlier  class.  The  additional  time  added  to  the  class  is  focused  on 
the  Data  Quality  Commander’ s  Statement.  The  addition  of  the  Service  Desk  Guides 
mentioned  above  in  the  response  to  Recommendation  #1  (a)  also  improves  the 
capabilities  and  skills  of  the  personnel  that  are  completing  the  Review  List  and  Data 
Quality  Commander’s  Statement.  The  Services  are  also  sponsoring  Data  Quality 
Conferences  to  increase  awareness  and  skill  level  of  the  personnel  completing  the 
various  data  quality  documents.  With  the  improvement  in  awareness  and  skill  levels 
greater  accountability  will  follow. 


RECOMMENDATION  4,;  We  recommend  that  the  military  treatment  facilities  comply  with 
the  policy  memorandum  guidance  pertaining  to  the  Data  Quality  Management  Control  Program 
by  accurately  completing  the  Data  Quality  Management  Control  Review  List  and  the  Data 
Quality  Statement. 

DoD  RESPONSE;  The  TRICARE  Management  Activity  (TMA)  concurs  with  this 
recommendation.  As  stated  above  in  the  subject  report,  the  October  17, 2001  Data  Quality 
Memorandum  signed  by  Mr.  Carrato  outlined  the  changes  in  the  Review  List,  Data  Quality 
Commander’s  Statmnent,  and  program  compliance  which  affect  this  line  item.  This  memo  was 
the  result  of  interim  input  from  DoDIG  visits  to  major  medical  centers  in  the  three  Services.  Mr. 
Carrato  outlined  three  major  changes  in  the  program  to  improve  MTF  compliance  with  the 
program.  These  were: 

a.  Changes  in  the  Review  List  and  Data  Quality  Commander’s  Statement, 

b.  Reporting  MTF  level  performance  to  TMA, 
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Final  Report 
Reference 


C.  And  inclusicm  of  DQMC  Program  into  Service  Medical  IG  and  /or  Audit  Agency 
compliance  program. 

It  should  be  noted  as  well  that  th^  are  many  MTF  success  stories  in  this  program. 


TECHNICAL  CHANGES;  Page  ii  and  4.  Results  (Page  ii)  and  Data  Quality  Management 
Omtrol  Program  for  the  Military  Health  System  (Page  4).  “As  a  result,  the  data  used  to  support 
the  Military  Retirement  Health  Benefit  Liability  estimate  continues  to  unreliable  and  the  DoD 

will  not  be  able  to  make  sound  management  decisions.”  Comment:  This  statement  is  too 
extensive  considering  the  scope  of  the  audit  conducted.  Recommend  that  the  sentence  be 
modified  by  adding:  “As  a  result,  there  are  still  concerns  on  the  data  used  to  support  the  Military 
Retirement  Health  Benefit  Liability  Estimate,  which  will  impact  DoDs  decision-making.” 


Revised 
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